The invention is directed to communications networks in general and to managing data loss in such networks. The invention has application in but is not limited to improving the distribution of critical data, such as is used in financial services, security or emergency services.
In today's financial markets, obtaining timely and accurate information is essential to performance. Financial institutions rely on high-speed, high-reliability networks to provide them with up-to-date news and price data on which trading decisions may be based. Market data delivery services have adopted a variety of different approaches. Typically, such data delivery services have been implemented using either TCP or unicast UDP. One problem with such implementations is limited scalability. Using TCP requires a separate socket (or end-point) and sliding window (or buffer) on the server for each recipient as well as a separate network stream for each client. UDP unicast requires a separate copy of the stream for each destination system. Both of these methods can quickly use up resources of the servers and the network to the point where data can be delayed for periods which are considered unacceptable. Consequently, UDP over IP multicast has been adopted by many stock exchanges to broadcast market information to subscribers and is widely considered as the only way to scale market data delivery. However, IP multicast, not being a connection-oriented mechanism, is unreliable and has no inherent mechanism for dealing with dropped packets. This may not be a problem in video or audio distribution but, for distributing critical data, a mechanism is required for efficiently managing data loss.
The invention provides for reliable communications while providing low-latency. In particular, a method is proposed of reliably transferring data via a plurality of channels in a network from a first node to a second node; in which the data comprises a plurality of parts; in which the channels comprise an unreliable channel and a reliable channel; the method including: sending the data to the second node via the unreliable channel; sending the data to the second node via the reliable channel; identifying parts of the data missing from the data received at the second node via the unreliable channel; obtaining from the data received at the second node via the reliable channel data parts corresponding to the data parts missing from the data received at the second node via the unreliable channel.
The invention preferably combines the low latency of RTP transport with the reliability of TCP and preferably provides a method in which an unreliable channel carries the data in a plurality of packets according to RTP operating over UDP; and a reliable, channel carries the data in a plurality of packets according to TCP. According to one variant, the unreliable channel operates RTP operating over UDP over multicast to a plurality of second nodes; and the reliable channel operates according to TCP to each of the plurality of second nodes. The data carried by each TCP packet preferably matches the data and RTP header carried by a corresponding one of the RTP packets.
Particular benefits derive from various detailed arrangements covering a plurality of second nodes. According to a further aspect of the invention, the unreliable channel operates according to RIP operating over UDP over multicast to a plurality N of second nodes; and the reliable channel operates over TCP to a subset M<N of the plurality N of second nodes; in which the plurality N of second nodes participate in a peer-to-peer network; in which the subset M of the second nodes receive via the reliable channel from the first node the data parts missing from the data received via the unreliable channel; and in which the remainder N-M of the plurality N of second nodes receive from one or other of the subset M of the second nodes the data parts missing from the data received via the unreliable channel.
Alternatively, the unreliable channel operates according to RTP operating over UDP over unicast to a plurality of second nodes and in which the first node and each second node participates in a peer-to-peer network used to receive from the first node the data parts missing from the data received via the unreliable channel.
According to a further aspect, the invention provides a plurality N of second nodes in which an unreliable channel operating according to RTP over UDP, is provided for each of the plurality N of second nodes and a reliable channel, operating according to TCP, is provided for each second node of a subset M<N of the plurality N of second nodes; in which the plurality N of second nodes participate in a multicast group and in which the subset M<N of the second nodes relay via the multicast group to a remainder N-M of the second nodes that are not included in the subset M<N the packets received via the reliable channels. The multicast group may use reliable multicast.
According to a further aspect, the invention includes merging the data received at the second node via the unreliable channel and the data received at the second node via the reliable channel into a single stream; identifying from the single stream the parts of the data missing from the data received at the second node via the unreliable channel; and obtaining from the single stream the data parts received at the second node via the reliable channel corresponding to the data parts missing from the data received at the second node via the unreliable channel.
The invention further provides a communications network for low-latency data transport that enjoys enhanced reliability, in particular, a communications network is provided comprising a first node and a second node; in which the first node is arranged to send data to the second node via two channels; a first unreliable channel for sending the data to the second node according to an unreliable communications protocol; and a second reliable channel for sending the data to the second node according to a reliable communications protocol; in which the data comprises a plurality of parts; in which the second node comprises a data checker for identifying data parts missing from the data received from the first node via the unreliable channel; and for obtaining from the data received from the first node via the reliable channel the data parts missing from the data received via the unreliable channel.
The invention may exploit a sequence numbering scheme to allow detection of missing data parts, in particular, in a communications network in which the data sent via the unreliable channel comprises a first series of packets in which each packet of the first series is associated with a sequence number designating a position in a sequence; in which the missing data is identified with reference to the sequence numbers. In the communications network, the data sent via the reliable channel may also comprise a second series of packets; in which each packet of the second series is associated with a sequence number designating a position in a sequence; in which each packet of the second series corresponds to one of the packets of the first series; and in which each packet of the second series is associated with the same sequence number as the corresponding packet of the first series.